{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using VMLS\n",
    "using LinearAlgebra"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 19\n",
    "# Constrained nonlinear least squares\n",
    "### 19.1 Constrained nonlinear least squares\n",
    "### 19.2 Penalty algorithm\n",
    "Let’s implement the penalty algorithm (algorithm [19.1](https://web.stanford.edu/~boyd/vmls/vmls.pdf#algorithmctr.19.1) in VMLS)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "penalty_method (generic function with 1 method)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function penalty_method(f, Df, g, Dg, x1, lambda1; kmax = 100,\n",
    "        feas_tol = 1e-4, oc_tol = 1e-4)\n",
    "    x = x1\n",
    "    mu = 1.0\n",
    "    feas_res = [norm(g(x))]\n",
    "    oc_res = [norm(2*Df(x)'*f(x) + 2*mu*Dg(x)'*g(x))]\n",
    "    lm_iters = zeros(Int64,0,1);\n",
    "    for k=1:kmax\n",
    "        F(x) = [f(x); sqrt(mu)*g(x)]\n",
    "        DF(x) = [Df(x); sqrt(mu)*Dg(x)]\n",
    "        x, hist = levenberg_marquardt(F,DF,x,lambda1,tol=oc_tol)\n",
    "        feas_res = [feas_res; norm(g(x))]\n",
    "        oc_res = [oc_res; hist[\"residuals\"][end]]\n",
    "        lm_iters = [lm_iters; length(hist[\"residuals\"])]\n",
    "        if norm(g(x)) < feas_tol\n",
    "            break\n",
    "        end\n",
    "        mu = 2*mu\n",
    "    end\n",
    "    return x, Dict([ (\"lm_iterations\", lm_iters),\n",
    "            (\"feas_res\", feas_res), (\"oc_res\", oc_res) ])\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On line $11$ we call the function `levenberg_marquardt` of the previous chapter to minimize $‖F(x)‖^2$ where\n",
    "\n",
    "$$\n",
    "F (x) =\n",
    "\\begin{bmatrix}\n",
    "f(x)\\\\\n",
    "\\sqrt{µ}g(x)\n",
    "\\end{bmatrix}\n",
    ".\n",
    "$$\n",
    "\n",
    "We evaluate two residuals. The “feasibility” residual $‖g(x(k))‖$ is the error in the constraint $g(x) = 0$. The “optimality condition” residual is defined as\n",
    "\n",
    "$$\n",
    "‖2Df(x^{(k)})^T f(x^{(k)}) + 2Dg(x^{(k)})^T z^{(k)}‖\n",
    "$$\n",
    "\n",
    "where $z(k) = 2µ(k−1)g(x(k))$ (and we take $µ(0) = µ(1)$). On line $13$, we obtain the optimality condition residual as the last residual in the Levenberg–Marquardt method. On line $20$ we return the final $x$, and a dictionary containing the two sequences of residuals and the number of iterations used in each call to the Levenberg–Marquardt algorithm.\n",
    "\n",
    "**Example.** We apply the method to a problem with two variables\n",
    "\n",
    "$$\n",
    "f(x_1, x_2) =\n",
    "\\begin{bmatrix}\n",
    "x_1 + exp(−x_2) \\\\\n",
    "x^2_1 + 2x_2 + 1\n",
    "\\end{bmatrix}\n",
    ", \n",
    "g(x_1, x_2) = x^2_1 + x^3_1 + x_2 + x^2_2.\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Float64,1}:\n",
       " -3.334955140841332e-5 \n",
       " -2.7682497163944097e-5"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(x) = [ x[1] + exp(-x[2]), x[1]^2 + 2*x[2] + 1 ];\n",
    "Df(x) = [ 1.0 -exp(-x[2]); 2*x[1] 2 ];\n",
    "g(x) = [ x[1] + x[1]^3 + x[2] + x[2]^2 ];\n",
    "Dg(x) = [ 1 + 3*x[1]^2 1 + 2*x[2] ];\n",
    "x, hist = penalty_method(f, Df, g, Dg, [0.5, -0.5], 1.0);\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following lines create a staircase plot with the residuals versus the cumulative number of Levenberg–Marquardt iterations as in VMLS figure [19.4](https://web.stanford.edu/~boyd/vmls/vmls.pdf#figure.19.4). The result is in figure [19.1](#fig1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip9200\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip9200)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip9201\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip9200)\" d=\"\n",
       "M279.82 1425.62 L2352.76 1425.62 L2352.76 47.2441 L279.82 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip9202\">\n",
       "    <rect x=\"279\" y=\"47\" width=\"2074\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  338.488,1425.62 338.488,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  628.207,1425.62 628.207,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  917.925,1425.62 917.925,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1207.64,1425.62 1207.64,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1497.36,1425.62 1497.36,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1787.08,1425.62 1787.08,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2076.8,1425.62 2076.8,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,1275.62 2352.76,1275.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,1075.24 2352.76,1075.24 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,874.866 2352.76,874.866 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,674.491 2352.76,674.491 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,474.116 2352.76,474.116 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,273.741 2352.76,273.741 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,73.3657 2352.76,73.3657 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,1425.62 279.82,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  338.488,1425.62 338.488,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  628.207,1425.62 628.207,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  917.925,1425.62 917.925,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1207.64,1425.62 1207.64,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1497.36,1425.62 1497.36,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1787.08,1425.62 1787.08,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2076.8,1425.62 2076.8,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,1275.62 310.914,1275.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,1075.24 310.914,1075.24 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,874.866 310.914,874.866 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,674.491 310.914,674.491 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,474.116 310.914,474.116 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,273.741 310.914,273.741 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,73.3657 310.914,73.3657 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 338.488, 1479.62)\" x=\"338.488\" y=\"1479.62\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 628.207, 1479.62)\" x=\"628.207\" y=\"1479.62\">20</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 917.925, 1479.62)\" x=\"917.925\" y=\"1479.62\">40</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1207.64, 1479.62)\" x=\"1207.64\" y=\"1479.62\">60</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1497.36, 1479.62)\" x=\"1497.36\" y=\"1479.62\">80</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1787.08, 1479.62)\" x=\"1787.08\" y=\"1479.62\">100</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2076.8, 1479.62)\" x=\"2076.8\" y=\"1479.62\">120</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 146.839, 1299.34)\" x=\"146.839\" y=\"1299.34\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 200.364, 1271.93)\" x=\"200.364\" y=\"1271.93\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 1271.93)\" x=\"223.203\" y=\"1271.93\">5 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 146.839, 1098.97)\" x=\"146.839\" y=\"1098.97\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 200.364, 1071.56)\" x=\"200.364\" y=\"1071.56\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 1071.56)\" x=\"223.203\" y=\"1071.56\">4 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 146.839, 898.594)\" x=\"146.839\" y=\"898.594\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 200.364, 871.183)\" x=\"200.364\" y=\"871.183\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 871.183)\" x=\"223.203\" y=\"871.183\">3 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 146.839, 698.219)\" x=\"146.839\" y=\"698.219\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 200.364, 670.808)\" x=\"200.364\" y=\"670.808\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 670.808)\" x=\"223.203\" y=\"670.808\">2 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 146.839, 497.844)\" x=\"146.839\" y=\"497.844\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 200.364, 470.433)\" x=\"200.364\" y=\"470.433\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 470.433)\" x=\"223.203\" y=\"470.433\">1 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 169.679, 297.468)\" x=\"169.679\" y=\"297.468\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 270.058)\" x=\"223.203\" y=\"270.058\">0 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 169.679, 97.0933)\" x=\"169.679\" y=\"97.0933\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 69.6829)\" x=\"223.203\" y=\"69.6829\">1 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1316.29, 1559.48)\" x=\"1316.29\" y=\"1559.48\">Cumulative Levenberg--Marquardt iterations</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\">Residual</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  338.488,359.094 1164.19,359.094 1164.19,343.892 1352.5,343.892 1352.5,377.041 1482.88,377.041 1482.88,418.497 1584.28,418.497 1584.28,467.433 1671.19,467.433 \n",
       "  1671.19,521.538 1758.11,521.538 1758.11,578.63 1830.54,578.63 1830.54,637.308 1902.97,637.308 1902.97,696.802 1975.4,696.802 1975.4,756.707 2033.34,756.707 \n",
       "  2033.34,816.817 2091.28,816.817 2091.28,877.033 2149.23,877.033 2149.23,937.3 2207.17,937.3 2207.17,997.593 2250.63,997.593 2250.63,1057.9 2294.09,1057.9 \n",
       "  2294.09,1118.21 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"338.488\" cy=\"359.094\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"338.488\" cy=\"359.094\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1164.19\" cy=\"359.094\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1164.19\" cy=\"359.094\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1164.19\" cy=\"343.892\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1164.19\" cy=\"343.892\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1352.5\" cy=\"343.892\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1352.5\" cy=\"343.892\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1352.5\" cy=\"377.041\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1352.5\" cy=\"377.041\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1482.88\" cy=\"377.041\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1482.88\" cy=\"377.041\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1482.88\" cy=\"418.497\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1482.88\" cy=\"418.497\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1584.28\" cy=\"418.497\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1584.28\" cy=\"418.497\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1584.28\" cy=\"467.433\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1584.28\" cy=\"467.433\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1671.19\" cy=\"467.433\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1671.19\" cy=\"467.433\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1671.19\" cy=\"521.538\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1671.19\" cy=\"521.538\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1758.11\" cy=\"521.538\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1758.11\" cy=\"521.538\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1758.11\" cy=\"578.63\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1758.11\" cy=\"578.63\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1830.54\" cy=\"578.63\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1830.54\" cy=\"578.63\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1830.54\" cy=\"637.308\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1830.54\" cy=\"637.308\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1902.97\" cy=\"637.308\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1902.97\" cy=\"637.308\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1902.97\" cy=\"696.802\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1902.97\" cy=\"696.802\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1975.4\" cy=\"696.802\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1975.4\" cy=\"696.802\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1975.4\" cy=\"756.707\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1975.4\" cy=\"756.707\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2033.34\" cy=\"756.707\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2033.34\" cy=\"756.707\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2033.34\" cy=\"816.817\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2033.34\" cy=\"816.817\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2091.28\" cy=\"816.817\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2091.28\" cy=\"816.817\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2091.28\" cy=\"877.033\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2091.28\" cy=\"877.033\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2149.23\" cy=\"877.033\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2149.23\" cy=\"877.033\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2149.23\" cy=\"937.3\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2149.23\" cy=\"937.3\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2207.17\" cy=\"937.3\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2207.17\" cy=\"937.3\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2207.17\" cy=\"997.593\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2207.17\" cy=\"997.593\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2250.63\" cy=\"997.593\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2250.63\" cy=\"997.593\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2250.63\" cy=\"1057.9\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2250.63\" cy=\"1057.9\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1057.9\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1057.9\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1118.21\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1118.21\" r=\"14\"/>\n",
       "<polyline clip-path=\"url(#clip9202)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  338.488,86.2547 1164.19,86.2547 1164.19,1082.06 1352.5,1082.06 1352.5,1096.03 1482.88,1096.03 1482.88,1108.08 1584.28,1108.08 1584.28,1119.97 1671.19,1119.97 \n",
       "  1671.19,1202.43 1758.11,1202.43 1758.11,1386.61 1830.54,1386.61 1830.54,1162.24 1902.97,1162.24 1902.97,1247.79 1975.4,1247.79 1975.4,1324.58 2033.34,1324.58 \n",
       "  2033.34,1106.06 2091.28,1106.06 2091.28,1164.95 2149.23,1164.95 2149.23,1224.58 2207.17,1224.58 2207.17,1284.56 2250.63,1284.56 2250.63,1112.31 2294.09,1112.31 \n",
       "  2294.09,1171.66 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"338.488\" cy=\"86.2547\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"338.488\" cy=\"86.2547\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1164.19\" cy=\"86.2547\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1164.19\" cy=\"86.2547\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1164.19\" cy=\"1082.06\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1164.19\" cy=\"1082.06\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1352.5\" cy=\"1082.06\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1352.5\" cy=\"1082.06\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1352.5\" cy=\"1096.03\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1352.5\" cy=\"1096.03\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1482.88\" cy=\"1096.03\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1482.88\" cy=\"1096.03\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1482.88\" cy=\"1108.08\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1482.88\" cy=\"1108.08\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1584.28\" cy=\"1108.08\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1584.28\" cy=\"1108.08\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1584.28\" cy=\"1119.97\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1584.28\" cy=\"1119.97\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1671.19\" cy=\"1119.97\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1671.19\" cy=\"1119.97\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1671.19\" cy=\"1202.43\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1671.19\" cy=\"1202.43\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1758.11\" cy=\"1202.43\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1758.11\" cy=\"1202.43\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1758.11\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1758.11\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1830.54\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1830.54\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1830.54\" cy=\"1162.24\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1830.54\" cy=\"1162.24\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1902.97\" cy=\"1162.24\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1902.97\" cy=\"1162.24\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1902.97\" cy=\"1247.79\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1902.97\" cy=\"1247.79\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1975.4\" cy=\"1247.79\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1975.4\" cy=\"1247.79\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1975.4\" cy=\"1324.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1975.4\" cy=\"1324.58\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2033.34\" cy=\"1324.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2033.34\" cy=\"1324.58\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2033.34\" cy=\"1106.06\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2033.34\" cy=\"1106.06\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2091.28\" cy=\"1106.06\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2091.28\" cy=\"1106.06\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2091.28\" cy=\"1164.95\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2091.28\" cy=\"1164.95\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2149.23\" cy=\"1164.95\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2149.23\" cy=\"1164.95\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2149.23\" cy=\"1224.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2149.23\" cy=\"1224.58\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2207.17\" cy=\"1224.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2207.17\" cy=\"1224.58\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2207.17\" cy=\"1284.56\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2207.17\" cy=\"1284.56\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2250.63\" cy=\"1284.56\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2250.63\" cy=\"1284.56\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2250.63\" cy=\"1112.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2250.63\" cy=\"1112.31\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1112.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1112.31\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1171.66\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9202)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1171.66\" r=\"14\"/>\n",
       "<path clip-path=\"url(#clip9200)\" d=\"\n",
       "M1818.67 312.204 L2280.76 312.204 L2280.76 130.764 L1818.67 130.764  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1818.67,312.204 2280.76,312.204 2280.76,130.764 1818.67,130.764 1818.67,312.204 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1842.67,191.244 1986.67,191.244 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip9200)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1926.67\" cy=\"191.244\" r=\"25\"/>\n",
       "<circle clip-path=\"url(#clip9200)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1926.67\" cy=\"191.244\" r=\"21\"/>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2010.67, 208.744)\" x=\"2010.67\" y=\"208.744\">Feasibility</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip9200)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1842.67,251.724 1986.67,251.724 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip9200)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1926.67\" cy=\"251.724\" r=\"25\"/>\n",
       "<circle clip-path=\"url(#clip9200)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1926.67\" cy=\"251.724\" r=\"21\"/>\n",
       "<g clip-path=\"url(#clip9200)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2010.67, 269.224)\" x=\"2010.67\" y=\"269.224\">Opt. cond.</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Plots\n",
    "cum_lm_iters = cumsum(hist[\"lm_iterations\"], dims=1);\n",
    "itr = vcat([0], [[i; i] for i in cum_lm_iters]...)\n",
    "feas_res = vcat([\n",
    "        [r;r] for r in hist[\"feas_res\"][1:end-1]]...,\n",
    "    hist[\"feas_res\"][end]);\n",
    "oc_res = vcat([[r;r] for r in hist[\"oc_res\"][1:end-1]]...,\n",
    "    hist[\"oc_res\"][end]);\n",
    "plot(itr, feas_res, shape=:circle, label = \"Feasibility\")\n",
    "plot!(itr, oc_res, shape=:circle, label = \"Opt. cond.\")\n",
    "plot!(yscale = :log10,\n",
    "    xlabel = \"Cumulative Levenberg--Marquardt iterations\",\n",
    "    ylabel = \"Residual\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 19.1** <a ind=\"#fig1\"></a> Feasibility and optimality condition errors versus the cumulative number of Levenberg–Marquardt iterations in the penalty algorithm.\n",
    "\n",
    "### 19.3 Augmented Lagrangian algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "aug_lag_method (generic function with 1 method)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function aug_lag_method(f, Df, g, Dg, x1, lambda1; kmax = 100,\n",
    "        feas_tol = 1e-4, oc_tol = 1e-4)\n",
    "    x = x1\n",
    "    z = zeros(length(g(x)))\n",
    "    mu = 1.0\n",
    "    feas_res = [norm(g(x))]\n",
    "    oc_res = [norm(2*Df(x)'*f(x) + 2*mu*Dg(x)'*z)]\n",
    "    lm_iters = zeros(Int64,0,1);\n",
    "    for k=1:kmax\n",
    "        F(x) = [f(x); sqrt(mu)*(g(x) + z/(2*mu))]\n",
    "        DF(x) = [Df(x); sqrt(mu)*Dg(x)]\n",
    "        x, hist = levenberg_marquardt(F, DF, x, lambda1, tol=oc_tol)\n",
    "        z = z + 2*mu*g(x)\n",
    "        feas_res = [feas_res; norm(g(x))]\n",
    "        oc_res = [oc_res; hist[\"residuals\"][end]]\n",
    "        lm_iters = [lm_iters; length(hist[\"residuals\"])]\n",
    "        if norm(g(x)) < feas_tol\n",
    "            break\n",
    "        end\n",
    "        mu = (norm(g(x)) < 0.25*feas_res[end-1]) ? mu : 2*mu\n",
    "    end\n",
    "    return x, z, Dict([ (\"lm_iterations\", lm_iters),\n",
    "            (\"feas_res\", feas_res), (\"oc_res\", oc_res)])\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the call to the Levenberg–Marquardt algorithm on line $12$ is to minimize\n",
    "$‖F (x)‖2$ where\n",
    "\n",
    "$$\n",
    "F (x) =\n",
    "\\begin{bmatrix}\n",
    "f(x)\\\\\n",
    "\\sqrt{µ^{(k)}}(g(x) + \\frac{z^{(k)}}{2µ^{(k)}}\n",
    "\\end{bmatrix}\n",
    ".\n",
    "$$\n",
    "\n",
    "We again record the feasibility residuals $‖g(x^{(k)}‖$ and the optimality conditions\n",
    "residuals\n",
    "\n",
    "$$\n",
    "‖2Df(x^{(k)})^T f(x^{(k)}) + 2Dg(x^{(k)})^T z^{(k)}‖,\n",
    "$$\n",
    "\n",
    "and return them in a dictionary.\n",
    "\n",
    "**Example.** We continue the small example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Float64,1}:\n",
       " -1.8646614856169702e-5\n",
       " -1.5008567819930016e-5"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x, z, hist = aug_lag_method(f, Df, g, Dg, [0.5, -0.5], 1.0);\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1-element Array{Float64,1}:\n",
       " -1.9999581273499105"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows the convergence as in VMLS figure [19.4](https://web.stanford.edu/~boyd/vmls/vmls.pdf#figure.19.4). The plot is\n",
    "given in figure [19.2](#fig2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip9600\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip9600)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip9601\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip9600)\" d=\"\n",
       "M279.82 1425.62 L2352.76 1425.62 L2352.76 47.2441 L279.82 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip9602\">\n",
       "    <rect x=\"279\" y=\"47\" width=\"2074\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  338.488,1425.62 338.488,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  787.02,1425.62 787.02,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1235.55,1425.62 1235.55,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1684.08,1425.62 1684.08,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2132.62,1425.62 2132.62,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,1271.92 2352.76,1271.92 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,1029.41 2352.76,1029.41 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,786.908 2352.76,786.908 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,544.404 2352.76,544.404 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,301.9 2352.76,301.9 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.82,59.3956 2352.76,59.3956 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,1425.62 279.82,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  338.488,1425.62 338.488,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  787.02,1425.62 787.02,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1235.55,1425.62 1235.55,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1684.08,1425.62 1684.08,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2132.62,1425.62 2132.62,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,1271.92 310.914,1271.92 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,1029.41 310.914,1029.41 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,786.908 310.914,786.908 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,544.404 310.914,544.404 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,301.9 310.914,301.9 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.82,59.3956 310.914,59.3956 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 338.488, 1479.62)\" x=\"338.488\" y=\"1479.62\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 787.02, 1479.62)\" x=\"787.02\" y=\"1479.62\">25</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1235.55, 1479.62)\" x=\"1235.55\" y=\"1479.62\">50</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1684.08, 1479.62)\" x=\"1684.08\" y=\"1479.62\">75</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2132.62, 1479.62)\" x=\"2132.62\" y=\"1479.62\">100</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 146.839, 1295.64)\" x=\"146.839\" y=\"1295.64\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 200.364, 1268.23)\" x=\"200.364\" y=\"1268.23\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 1268.23)\" x=\"223.203\" y=\"1268.23\">4 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 146.839, 1053.14)\" x=\"146.839\" y=\"1053.14\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 200.364, 1025.73)\" x=\"200.364\" y=\"1025.73\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 1025.73)\" x=\"223.203\" y=\"1025.73\">3 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 146.839, 810.635)\" x=\"146.839\" y=\"810.635\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 200.364, 783.225)\" x=\"200.364\" y=\"783.225\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 783.225)\" x=\"223.203\" y=\"783.225\">2 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 146.839, 568.131)\" x=\"146.839\" y=\"568.131\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 200.364, 540.721)\" x=\"200.364\" y=\"540.721\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 540.721)\" x=\"223.203\" y=\"540.721\">1 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 169.679, 325.627)\" x=\"169.679\" y=\"325.627\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 298.217)\" x=\"223.203\" y=\"298.217\">0 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 169.679, 83.1232)\" x=\"169.679\" y=\"83.1232\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:38px; text-anchor:start;\" transform=\"rotate(0, 223.203, 55.7128)\" x=\"223.203\" y=\"55.7128\">1 </text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1316.29, 1559.48)\" x=\"1316.29\" y=\"1559.48\">Cumulative Levenberg-Marquardt iterations</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\">Residual</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  338.488,405.199 1361.14,405.199 1361.14,386.8 1666.14,386.8 1666.14,489.133 1827.61,489.133 1827.61,637.996 1971.14,637.996 1971.14,789.695 2078.79,789.695 \n",
       "  2078.79,942.371 2168.5,942.371 2168.5,1093.58 2240.26,1093.58 2240.26,1251.33 2294.09,1251.33 2294.09,1386.61 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"338.488\" cy=\"405.199\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"338.488\" cy=\"405.199\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1361.14\" cy=\"405.199\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1361.14\" cy=\"405.199\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1361.14\" cy=\"386.8\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1361.14\" cy=\"386.8\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.14\" cy=\"386.8\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1666.14\" cy=\"386.8\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.14\" cy=\"489.133\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1666.14\" cy=\"489.133\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1827.61\" cy=\"489.133\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1827.61\" cy=\"489.133\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1827.61\" cy=\"637.996\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1827.61\" cy=\"637.996\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1971.14\" cy=\"637.996\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1971.14\" cy=\"637.996\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1971.14\" cy=\"789.695\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1971.14\" cy=\"789.695\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2078.79\" cy=\"789.695\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2078.79\" cy=\"789.695\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2078.79\" cy=\"942.371\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2078.79\" cy=\"942.371\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2168.5\" cy=\"942.371\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2168.5\" cy=\"942.371\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2168.5\" cy=\"1093.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2168.5\" cy=\"1093.58\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2240.26\" cy=\"1093.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2240.26\" cy=\"1093.58\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2240.26\" cy=\"1251.33\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2240.26\" cy=\"1251.33\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1251.33\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1251.33\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1386.61\" r=\"14\"/>\n",
       "<polyline clip-path=\"url(#clip9602)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  338.488,86.2547 1361.14,86.2547 1361.14,1280.17 1666.14,1280.17 1666.14,1287.55 1827.61,1287.55 1827.61,1283.86 1971.14,1283.86 1971.14,1349.72 2078.79,1349.72 \n",
       "  2078.79,1301.52 2168.5,1301.52 2168.5,1316.79 2240.26,1316.79 2240.26,1364.75 2294.09,1364.75 2294.09,1330.74 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"338.488\" cy=\"86.2547\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"338.488\" cy=\"86.2547\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1361.14\" cy=\"86.2547\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1361.14\" cy=\"86.2547\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1361.14\" cy=\"1280.17\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1361.14\" cy=\"1280.17\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.14\" cy=\"1280.17\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1666.14\" cy=\"1280.17\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.14\" cy=\"1287.55\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1666.14\" cy=\"1287.55\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1827.61\" cy=\"1287.55\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1827.61\" cy=\"1287.55\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1827.61\" cy=\"1283.86\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1827.61\" cy=\"1283.86\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1971.14\" cy=\"1283.86\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1971.14\" cy=\"1283.86\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1971.14\" cy=\"1349.72\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1971.14\" cy=\"1349.72\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2078.79\" cy=\"1349.72\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2078.79\" cy=\"1349.72\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2078.79\" cy=\"1301.52\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2078.79\" cy=\"1301.52\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2168.5\" cy=\"1301.52\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2168.5\" cy=\"1301.52\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2168.5\" cy=\"1316.79\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2168.5\" cy=\"1316.79\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2240.26\" cy=\"1316.79\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2240.26\" cy=\"1316.79\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2240.26\" cy=\"1364.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2240.26\" cy=\"1364.75\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1364.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1364.75\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1330.74\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9602)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2294.09\" cy=\"1330.74\" r=\"14\"/>\n",
       "<path clip-path=\"url(#clip9600)\" d=\"\n",
       "M1818.67 312.204 L2280.76 312.204 L2280.76 130.764 L1818.67 130.764  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1818.67,312.204 2280.76,312.204 2280.76,130.764 1818.67,130.764 1818.67,312.204 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1842.67,191.244 1986.67,191.244 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip9600)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1926.67\" cy=\"191.244\" r=\"25\"/>\n",
       "<circle clip-path=\"url(#clip9600)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1926.67\" cy=\"191.244\" r=\"21\"/>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2010.67, 208.744)\" x=\"2010.67\" y=\"208.744\">Feasibility</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip9600)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1842.67,251.724 1986.67,251.724 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip9600)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1926.67\" cy=\"251.724\" r=\"25\"/>\n",
       "<circle clip-path=\"url(#clip9600)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1926.67\" cy=\"251.724\" r=\"21\"/>\n",
       "<g clip-path=\"url(#clip9600)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 2010.67, 269.224)\" x=\"2010.67\" y=\"269.224\">Opt. cond.</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Plots\n",
    "cum_lm_iters = cumsum(hist[\"lm_iterations\"],dims=1);\n",
    "itr = vcat([0], [[i; i] for i in cum_lm_iters]...)\n",
    "feas_res = vcat([\n",
    "        [r;r] for r in hist[\"feas_res\"][1:end-1]]...,hist[\"feas_res\"][end]);\n",
    "oc_res = vcat([\n",
    "        [r;r] for r in hist[\"oc_res\"][1:end-1]]...,\n",
    "    hist[\"oc_res\"][end]);\n",
    "plot(itr, feas_res, shape=:circle, label = \"Feasibility\")\n",
    "plot!(itr, oc_res, shape=:circle, label = \"Opt. cond.\")\n",
    "plot!(yscale = :log10,\n",
    "    xlabel = \"Cumulative Levenberg-Marquardt iterations\",\n",
    "    ylabel = \"Residual\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 19.2** Feasibility and optimality condition errors versus the cumulative number of Levenberg–Marquardt iterations in the augmented Lagrangian algorithm.\n",
    "\n",
    "### 19.4 Nonlinear control"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.1.1",
   "language": "julia",
   "name": "julia-1.1"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
